Purpose

This PR has two related goals:

1. Refactor the global-index build pipeline so scan planning is shared by Core and execution-specific Flink/Spark builders consume an explicit scan result.
2. Add opt-in incremental refresh for data-evolution indexes when indexed values in already covered row ranges change.

Previously, generic and sorted index builders mixed scan planning, mutable build state, and index-file writing. Incremental builds also considered only uncovered row ranges, so an index could remain readable but become stale after partial updates to indexed columns.

Global-index build refactoring

• Introduce ScanResult, which carries the scan snapshot ID, selected row ranges, scan entries, and index entries to delete.
• Move generic index scan planning into Core as GenericGlobalIndexScanner and remove the Flink-specific GenericGlobalIndexBuilder.
• Split the sorted builder into SortedGlobalIndexScanner and SortedGlobalIndexWriter so scan state is returned explicitly instead of retained in mutable builder fields.
• Move shared row-range calculation, split grouping, and shard helpers into GlobalIndexBuilderUtils.
• Make generic and sorted scanners expose consistent scan and incrementalScan flows.
• Adapt the Flink and Spark topology builders to consume ScanResult and keep index writing separate from scan planning.
• Remove test-only overloads and duplicate incremental-range planning from execution modules.

Data-evolution index refresh

• Record the data snapshot scanned by an index build as scanSnapshotId in GlobalIndexMeta.sourceMeta, encoded with a DEIX magic number and version.
• Record the scan snapshot rather than the later index-commit snapshot, so updates committed while a build is running remain newer than the watermark and are found by the next build.
• When global-index.detect-datafile-change=true, select an existing index for refresh when an active data file:
  • belongs to the same partition and bucket,
  • intersects the index row range,
  • physically contains at least one indexed field, resolved by stable field ID, and
  • has maxSequenceNumber greater than the index's scanSnapshotId.
• Merge affected indexed ranges with newly uncovered ranges for generic and sorted indexes, including vector, BTree, bitmap, and full-text implementations.
• Commit deletion of old index files and addition of their replacements atomically, so readers never observe an index-unavailable gap.
• Reject deletion of a missing global-index file by default to detect concurrent replacement. global-index.ignore-missing-delete=true enables permissive behavior when explicitly required.

Data-file change detection is disabled by default because it requires scanning manifest entries. Existing uncovered-range incremental builds retain their previous behavior unless the option is enabled.

Compatibility and scope

Indexes created before DEIX source metadata was introduced do not have a trustworthy scan watermark. This PR intentionally does not perform automatic migration or a one-time refresh for those indexes. They remain treated as covered and must be explicitly rebuilt once to populate sourceMeta before participating in change-based refresh.

User impact

For data-evolution tables with change detection enabled, users can continuously update indexed columns and run incremental index builds. Unaffected ranges remain available, while affected ranges are rebuilt and atomically replace their previous index files. Updates to unrelated physical columns do not trigger a refresh.

Tests

• Source-metadata serialization, magic/version validation, and scan-snapshot propagation.
• Generic and sorted scanner behavior, writer lifecycle, row-range splitting, and shard planning.
• Refresh planning across sequence watermarks, physical field projections, schema renames, multiple indexed fields, partitions, buckets, and overlapping ranges.
• Legacy-index exclusion and no-op incremental builds when no indexed value changed.
• Atomic replacement, strict missing-delete handling, and configurable permissive behavior.
• Flink and Spark coverage for vector, BTree, bitmap, and full-text index build paths.
• Lumina native integration coverage on Linux.